Projecting multi-faceted image onto convex polyhedron based on wide-angle image

ABSTRACT

Systems and methods for projecting a multi-faceted image onto a convex polyhedron based on an input image are described. A system can include a controller configured to determine a mapping between pixels within a wide-angle image and a multi-faceted image, and generate the multi-faceted image based on the mapping.

INTRODUCTION

The technical field generally relates to generating an image, and moreparticularly projecting a multi-faceted image onto a convex polyhedronbased on an input image.

Vehicles (e.g., automobiles, farm equipment, automated factoryequipment, construction equipment) increasingly include sensor systemsthat facilitate augmented or automated actions. A camera (e.g., still,video) facilitates target classification (e.g., pedestrian, truck, tree)using a neural network processor, for example. In autonomous driving,sensors should cover three hundred and sixty-degrees around the vehicle.However, different types of sensors obtain different types ofinformation in different coordinate spaces.

SUMMARY

A system for projecting a multi-faceted image on a convex polyhedron isprovided. The system can include a controller configured to determine amapping between pixels within a wide-angle image and a multi-facetedimage, and generate the multi-faceted image based on the mapping.

In other features, each facet of the multi-faceted image comprises aplurality of pixels from a subset of the wide-angle image.

In other features, the controller is further configured to receive thewide-angle image from a camera.

In other features, the system includes the camera.

In other features, the camera comprises a fisheye camera.

In other features, the controller is further configured to determine themapping based according to a standard fish-eye equidistant model:

u _(F) =u _(F,0) +c _(F)θcos ϕ, and

v _(F) =v _(F,0) +c _(F)θsin ϕ,

-   -   where u_(F) and v_(F) are pixel locations of the pixels within        the wide-angle image, c_(F), u_(F,0) and v_(F,0) are intrinsic        camera parameters, θ=arccos(n_(F,Z)) and        ϕ=arctan2(n_(F,Y),n_(F,X)), where n_(F,X), n_(F,Y), and n_(F,Z)        comprise a respective pixel unit vector corresponding to a        Cartesian coordinate system.

In other features, the multi-faceted image includes at least a firstfacet, a second facet, and a third facet.

A method for projecting a multi-faceted image on a convex polyhedron isdisclosed. The method can include determining, by a controller, amapping between pixels within a wide-angle image and a multi-facetedimage, and generating, by the controller, the multi-faceted image basedon the mapping.

In other features, each facet of the multi-faceted image comprises aplurality of pixels from a subset of the wide-angle image.

In other features, the method includes receiving the wide-angle imagefrom a camera.

In other features, the camera comprises a fisheye camera.

In other features, the mapping is determining according to:

u _(F) =u _(F,0) +c _(F)θcos ϕ, and

v _(F) =v _(F,0) +c _(F)θsin ϕ,

-   -   where u_(F) and v_(F) are pixel locations of the pixels within        the wide-angle image, c_(F), u_(F,0) and v_(F,0) are intrinsic        camera parameters, θ=arccos(n_(F,Z)) and        ϕ=arctan2(n_(F,Y),n_(F,X)), where n_(F,X), n_(F,Y), and n_(F,Z)        comprise a respective pixel unit vector corresponding to a        Cartesian coordinate system.

In other features, the multi-faceted image includes at least a firstfacet, a second facet, and a third facet.

In other features, the multi-faceted image comprises a multi-facetedpiece-wise projective image.

A vehicle including a system for projecting a multi-faceted image on aconvex polyhedron is disclosed. The system can include a controllerconfigured to determine a mapping between pixels within a wide-angleimage and a multi-faceted image, and generate the multi-faceted imagebased on the mapping.

In other features, each facet of the multi-faceted image comprises aplurality of pixels from a subset of the wide-angle image.

In other features, the controller is further configured to receive thewide-angle image from a camera.

In other features, the system includes the camera.

In other features, the camera comprises a fisheye camera.

In other features, the controller is further configured to determine themapping based according to:

u _(F) =u _(F,0) +c _(F)θcos ϕ, and

v _(F) =v _(F,0) +c _(F)θsin ϕ,

-   -   where u_(F) and v_(F) are pixel locations of the pixels within        the wide-angle image, c_(F), u_(F,0) and ν_(F,0) are intrinsic        camera parameters, θ=arccos(n_(F,Z)) and        ϕ=arctan2(n_(F,Y),n_(F,X)), where n_(F,X), n_(F,Y), and n_(F,Z)        comprise a respective pixel unit vector corresponding to a        Cartesian coordinate system.

In other features, the multi-faceted image includes at least a firstfacet, a second facet, and a third facet.

BRIEF DESCRIPTION OF THE DRAWINGS

The exemplary implementations will hereinafter be described inconjunction with the following drawing figures, wherein like numeralsdenote like elements, and wherein:

FIG. 1 is a schematic illustration of a vehicle including a controllerconfigured to project a multi-faceted image on a convex polyhedron basedon an input image;

FIG. 2 is a schematic illustration of a vehicle including a wide-angleview camera;

FIG. 3 is an example image captured by the wide-angle camera;

FIG. 4 is a schematic illustration of a unit sphere having a unit cubeprojected therein;

FIG. 5 is a perspective view of the unit sphere having the unit cubeprojected therein, where portions of the unit cube correspond to facetsof an image;

FIG. 6 is another perspective view of the unit sphere having the unitcube projected therein, where portions of the unit cube correspond tofacets of an image;

FIG. 7 is an example image captured by a wide-angle camera;

FIG. 8 is an example generated multi-faceted image generated based onthe image shown in FIG. 7 ;

FIG. 9 is an example generated multi-faceted image generated based onthe image shown in FIG. 7 ; and

FIG. 10 is a flow diagram illustrating an example process for generatinga multi-faceted image based on an input image.

DETAILED DESCRIPTION

With reference to FIG. 1 , a vehicle 10 is shown in accordance withvarious implementations. The vehicle 10 generally includes a chassis 12,a body 14, front wheels 16, and rear wheels 18. The body 14 is arrangedon the chassis 12 and substantially encloses components of the vehicle10. The body 14 and the chassis 12 may jointly form a frame. The wheels16 and 18 are each rotationally coupled to the chassis 12 near arespective corner of the body 14.

In various implementations, the vehicle 10 is an autonomous vehicle. Theautonomous vehicle is, for example, a vehicle that is automaticallycontrolled to carry passengers from one location to another. The vehicle10 is depicted in the illustrated implementation as a passenger car, butit should be appreciated that any other vehicle including motorcycles,trucks, sport utility vehicles (SUVs), recreational vehicles (RVs),marine vessels, aircraft, etc., can also be used. In an exemplaryimplementation, the autonomous vehicle is an automation system of LevelTwo or higher. A Level Two automation system indicates “partialautomation”. However, in other implementations, the autonomous vehiclemay be a so-called Level Three, Level Four or Level Five automationsystem. A Level Three automation system indicates conditionalautomation. A Level Four system indicates “high automation”, referringto the driving mode-specific performance by an automated driving systemof all aspects of the dynamic driving task, even when a human driverdoes not respond appropriately to a request to intervene. A Level Fivesystem indicates “full automation”, referring to the full-timeperformance by an automated driving system of all aspects of the dynamicdriving task under all roadway and environmental conditions that can bemanaged by a human driver.

As shown, the vehicle 10 generally includes a propulsion system 20, atransmission system 22, a steering system 24, a brake system 26, asensor system 28, an actuator system 30, at least one data storagedevice 32, at least one controller 34, and a communication system 36.The propulsion system 20 may, in various implementations, include aninternal combustion engine, an electric machine such as a tractionmotor, and/or a fuel cell propulsion system. The transmission system 22is configured to transmit power from the propulsion system 20 to thevehicle wheels 16 an 18 according to selectable speed ratios. Accordingto various implementations, the transmission system 22 may include astep-ratio automatic transmission, a continuously-variable transmission,or other appropriate transmission. The brake system 26 is configured toprovide braking torque to the vehicle wheels 16 and 18. The brake system26 may, in various implementations, include friction brakes, brake bywire, a regenerative braking system such as an electric machine, and/orother appropriate braking systems. The steering system 24 influences aposition of the of the vehicle wheels 16 and 18. While depicted asincluding a steering wheel for illustrative purposes, in someimplementations contemplated within the scope of the present disclosure,the steering system 24 may not include a steering wheel.

The sensor system 28 includes one or more sensing devices 40 a- 40 nthat sense observable conditions of the exterior environment and/or theinterior environment of the vehicle 10. The sensing devices 40 a-40 ncan include, but are not limited to, radars, lidars, global positioningsystems, optical cameras, thermal cameras, ultrasonic sensors, and/orother sensors. The actuator system 30 includes one or more actuatordevices 42 a-42 n that control one or more vehicle features such as, butnot limited to, the propulsion system 20, the transmission system 22,the steering system 24, and the brake system 26. In variousimplementations, the vehicle features can further include interiorand/or exterior vehicle features such as, but are not limited to, doors,a trunk, and cabin features such as air, music, lighting, etc.

The communication system 36 is configured to wirelessly communicateinformation to and from other entities 48, such as but not limited to,other vehicles (“V2V” communication,) infrastructure (“V2I”communication), remote systems, and/or personal devices (described inmore detail with regard to FIG. 2 ). In an exemplary implementation, thecommunication system 36 is a wireless communication system configured tocommunicate via a wireless local area network (WLAN) using IEEE 802.11standards or by using cellular data communication. However, additionalor alternate communication methods, such as a dedicated short-rangecommunications (DSRC) channel, are also considered within the scope ofthe present disclosure. DSRC channels refer to one-way or two-wayshort-range to medium-range wireless communication channels specificallydesigned for automotive use and a corresponding set of protocols andstandards.

The data storage device 32 stores data for use in automaticallycontrolling functions of the vehicle 10. In various implementations, thedata storage device 32 stores defined maps of the navigable environment.In various implementations, the defined maps may be predefined by andobtained from a remote system (described in further detail with regardto FIG. 2 ). For example, the defined maps may be assembled by theremote system and communicated to the vehicle 10 (wirelessly and/or in awired manner) and stored in the data storage device 32. As can beappreciated, the data storage device 32 may be part of the controller34, separate from the controller 34, or part of the controller 34 andpart of a separate system.

The controller 34 includes at least one processor 44 and a computerreadable storage device or media 46. The processor 44 can be any custommade or commercially available processor, a central processing unit(CPU), a graphics processing unit (GPU), an auxiliary processor amongseveral processors associated with the controller 34, asemiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, any combination thereof, or generally any devicefor executing instructions. The computer readable storage device ormedia 46 may include volatile and nonvolatile storage in read-onlymemory (ROM), random-access memory (RAM), and keep-alive memory (KAM),for example. KAM is a persistent or non-volatile memory that may be usedto store various operating variables while the processor 44 is powereddown. The computer-readable storage device or media 46 may beimplemented using any of a number of known memory devices such as PROMs(programmable read-only memory), EPROMs (electrically PROM), EEPROMs(electrically erasable PROM), flash memory, or any other electric,magnetic, optical, or combination memory devices capable of storingdata, some of which represent executable instructions, used by thecontroller 34 in controlling and executing functions of the vehicle 10.

The instructions may include one or more separate programs, each ofwhich comprises an ordered listing of executable instructions forimplementing logical functions. The instructions, when executed by theprocessor 34, receive and process signals from the sensor system 28,perform logic, calculations, methods and/or algorithms for automaticallycontrolling the components of the vehicle 10, and generate controlsignals to the actuator system 30 to automatically control thecomponents of the vehicle 10 based on the logic, calculations, methods,and/or algorithms. Although only one controller 34 is shown in FIG. 1 ,implementations of the vehicle 10 can include any number of controllers34 that communicate over any suitable communication medium or acombination of communication mediums and that cooperate to process thesensor signals, perform logic, calculations, methods, and/or algorithms,and generate control signals to automatically control features of thevehicle 10.

Referring to FIG. 2 , the sensing device 40 a can comprise camera 102that is configured to capture one or more images within a field-of-view(FOV) of the camera 102. In an example implementation, the camera 102comprises a fisheye camera that includes an ultra-wide-angle lens thatcaptures environments that result in images having a characteristicconvex non-rectilinear appearance. While illustrated as being orientedtowards a front end of the vehicle 10, it is understood that the camera102 may be positioned in one or more other suitable locations around thevehicle 10 to capture the surrounding environment. When the camera 102comprises a fisheye camera, the FOV of the camera 102 may be greaterthan or equal to one hundred and eighty degrees (≥180°). It isunderstood that the principles of this disclosure can also be applied toomnidirectional cameras, i.e., having a FOV up to three hundred andsixty degrees (360°).

As the vehicle 10 traverses an environment 100, the camera 102 maycapture one or more objects 104 within a FOV of the camera 102 (seeFIGS. 2 and 3 ). Due to the configuration of fisheye camera lenses,objects, such as a person, depicted in a resulting image may bedistorted as shown in FIG. 3 .

As discussed herein, the controller 34 can transform an image generatedby the camera 102 to a multi-faceted image. For example, the controller34 projects an image onto a convex polyhedron as discussed in greaterdetail below with respect to FIGS. 4 through 6 . In an exampleimplementation, the controller 34 receives an image, such as image 302shown in FIG. 3 , captured by the camera 102 that includes theultra-wide-angle lens. The controller 34 maps pixels within thewide-angle projective image to a multi-faceted piece-wise projectiveimage. For example, the multi-faceted piece-wise projective image caninclude multiple facets, i.e., portions, that, when taken together, formthe whole image.

FIG. 4 illustrates an example unit sphere 400 that represents the camera102 and a unit cube 402 projected therein. The unit sphere 400 includesa center 404 that represents a center of the camera 102 lens. The unitsphere 400 and the unit cube 402 can include x-, y-, and z-componentsthat are translations according to a Cartesian coordinate system(comprising an X-axis, a Y-axis, and a Z-axis).

As shown in FIGS. 5 and 6 , the unit cube 402 can represent an imagehaving multiple facets, and each facet can include a subset of pixelssuch that the generated image appears to have been captured by a camerahaving a narrower FOV than the fisheye camera. FIG. 5 illustrates animplementation in which the unit cube 402 includes three (3) facets 502,504, 506, and FIG. 6 illustrates an implementation in which the unitcube 402 includes two (2) facets 602, 604.

The controller 34 uses suitable image processing techniques to generatemultiple facets of an image based on the image captured by the camera102. The controller 34 can transform pixels captured by the camera 102,e.g., the fisheye camera, to corresponding facets to generate themulti-faceted image. The controller 34 can begin the transformationprocess by normalizing a pixel direction according to Equation 1:

$\begin{matrix}{{ \lbrack {u,v,f} \rbrack_{f}^{T}arrow\frac{\lbrack {u,v,f} \rbrack_{f}^{T}}{\sqrt{u^{2} + v^{2} + f^{2}}}  = n_{k}},} & {{Equation}1}\end{matrix}$

where the set [u,v]_(k) represents a camera 102 pixel location in thekth facet (where k is an integer greater than or equal to one such as afirst facet, a second facet, and so forth), f is a focal distance of thecamera 102, T is a transpose for a column vector, and n_(k) is a pixelunit vector for the kth facet. The controller 34 can then rotate thepixel unit vector (for a corresponding pixel having a Cartesiancoordinate) according to Equation 2:

n_(F)=R_(Fk)n_(k)  Equation 2,

where R_(Fk) is a rotation matrix that connects the fisheye cameraparameters to a desired camera parameter, i.e., camera having a narrowerFOV than the fisheye camera. For example, the rotation matrix caninclude transformation data that maps fisheye camera parameters to whichfacet to include the pixels in, etc. The controller 34 can thendetermine corresponding fisheye camera pixels according to Equations 3and 4:

u _(F) =u _(F,0) +c _(F)θcos ϕ  Equation 3,

v _(F) =v _(F,0) +c _(F)θsin ϕ  Equation 4,

where u_(F) and v_(F) are respective corresponding fisheye pixellocations, c_(F) and the set [u, v]_(F,0) are intrinsic fisheye cameraparameters, and θ=arccos(n_(F,Z)) and ϕ=arctan2(n_(F,Y), n_(F,X)). Theparameters n_(F,X), n_(F,Y), and n_(F,Z) comprise a respective pixelunit vector within the corresponding X-axis, Y-axis, and Z-axis withinthe Cartesian coordinate system. Using the corresponding fisheye pixellocations, the controller 34 generates a multi-faceted image based onthe determined mapping of fisheye camera pixels to the multi-facetedpiecewise-projective image pixels. For example, the controller 34inserts fisheye pixels using the location data into the correspondingCartesian coordinate system. The resulting generated image can includeless distortion with respect to the image generated by the fisheyecamera, e.g.,

Referring to FIGS. 7 through 9 , FIG. 7 illustrates another exampleimage 700 captured by the camera 102. FIG. 8 illustrates an examplegenerated image 800 that includes facets 802, 804, 806 based on theconfiguration discussed within respect to FIG. 5 , and FIG. 9illustrates an example generated image 900 that includes facets 902, 904based on the configuration discussed within respect to FIG. 6 .

FIG. 10 is a flow diagram is shown that illustrates an example process1000 of generating a multi-faceted image based on an image captured bythe camera 102, such as a fisheye camera. The process 1000 comprisessoftware instructions executable by the controller 34, e.g., theprocessor 44 and the computer readable storage device or media 46.

The process 1000 begins at block 1002 in which a determination is madewhether an image, such as a wide-angle projective image, has beenreceived. If an image has not been received, the process 1000 returns toblock 1002. Otherwise, the controller 34 normalizes the pixeldirection(s) at block 1004. At block 1006, the controller 34 rotates thepixel unit vector. At block 1008, the controller 34 determinescorresponding fisheye camera pixels. At block 1010, the controller 34generates the multi-faceted image based on the determined mappingbetween fisheye pixels to multi-faceted piecewise-projective imagepixels. The process 1000 then ends.

While at least one exemplary implementation has been presented in theforegoing detailed description, it should be appreciated that a vastnumber of variations exist. It should also be appreciated that theexemplary implementation or exemplary implementations are only examples,and are not intended to limit the scope, applicability, or configurationof the disclosure in any way. Rather, the foregoing detailed descriptionwill provide those skilled in the art with a convenient road map forimplementing the exemplary implementation or exemplary implementations.It should be understood that various changes can be made in the functionand arrangement of elements without departing from the scope of thedisclosure as set forth in the appended claims and the legal equivalentsthereof.

The detailed description is merely exemplary in nature and is notintended to limit the application and uses. Furthermore, there is nointention to be bound by any expressed or implied theory presented inthe preceding technical field, background, brief summary, or thefollowing detailed description. As used herein, the term module refersto any hardware, software, firmware, electronic control component,processing logic, and/or processor device, individually or in anycombination, including without limitation: application specificintegrated circuit (ASIC), an electronic circuit, a processor (shared,dedicated, or group) and memory that executes one or more software orfirmware programs, a combinational logic circuit, and/or other suitablecomponents that provide the described functionality.

Implementations of the present disclosure may be described herein interms of functional and/or logical block components and variousprocessing steps. It should be appreciated that such block componentsmay be realized by any number of hardware, software, and/or firmwarecomponents configured to perform the specified functions. For example,an implementation of the present disclosure may employ variousintegrated circuit components, e.g., memory elements, digital signalprocessing elements, logic elements, look-up tables, or the like, whichmay carry out a variety of functions under the control of one or moremicroprocessors or other control devices. In addition, those skilled inthe art will appreciate that implementations of the present disclosuremay be practiced in conjunction with any number of systems, and that thesystems described herein is merely exemplary implementations of thepresent disclosure.

What is claimed is:
 1. A system for projecting a multi-faceted image ona convex polyhedron based on an input image, the system comprising: acontroller configured to: determine a mapping between pixels within awide-angle image and a multi-faceted image; and generate themulti-faceted image based on the mapping.
 2. The system of claim 1,wherein each facet of the multi-faceted image comprises a plurality ofpixels from a subset of the wide-angle image.
 3. The system of claim 1,wherein the controller is further configured to receive the wide-angleimage from a camera.
 4. The system of claim 3, further comprising thecamera.
 5. The system of claim 4, wherein the camera comprises a fisheyecamera.
 6. The system of claim 1, wherein the controller is furtherconfigured to determine the mapping based according to:u _(F) =u _(F,0) +c _(F)θcos ϕ, andv _(F) =v _(F,0) +c _(F)θsin ϕ, where u_(F)and v_(F) are pixel locationsof the pixels within the wide-angle image, c_(F), u_(F,0) and v_(F,0)are intrinsic camera parameters, θ=arccos(n_(F,Z)) andϕ=arctan2(n_(F,Y),n_(F,X)), where n_(F,X), n_(F,Y), and n_(F,Z) comprisea respective pixel unit vector corresponding to a Cartesian coordinatesystem.
 7. The system of claim 1, wherein the multi-faceted imageincludes at least a first facet, a second facet, and a third facet.
 8. Amethod for projecting a multi-faceted image on a convex polyhedron, themethod comprising: determining, by a controller, a mapping betweenpixels within a wide-angle image and a multi-faceted image; andgenerating, by the controller, the multi-faceted image based on themapping.
 9. The method of claim 8, wherein each facet of themulti-faceted image comprises a plurality of pixels from a subset of thewide-angle image.
 10. The method of claim 8, the method furthercomprising: receiving the wide-angle image from a camera.
 11. The methodof claim 10, wherein the camera comprises a fisheye camera.
 12. Themethod of claim 8, wherein the mapping is determining according to:u _(F) =u _(F,0) +c _(F)θcos ϕ, andv _(F) =v _(F,0) +c _(F)θsin ϕ, where u_(F)and v_(F) are pixel locationsof the pixels within the wide-angle image, c_(F), u_(F,0) and v_(F,0)are intrinsic camera parameters, θ=arccos(n_(F,Z)) and ϕ=arctan2(n_(F,Y), n_(F,X)), where n_(F,X), n_(F,Y), and n_(F,Z) comprise arespective pixel unit vector corresponding to a Cartesian coordinatesystem.
 13. The method of claim 8, wherein the multi-faceted imageincludes at least a first facet, a second facet, and a third facet. 14.The method of claim 8, wherein the multi-faceted image comprises amulti-faceted piece-wise projective image.
 15. A vehicle, comprising asystem for projecting a multi-faceted image on a convex polyhedron basedon an input image, the system comprising: a controller, configured todetermine a mapping between pixels within a wide-angle image and amulti-faceted image; and generate the multi-faceted image based on themapping.
 16. The vehicle of claim 15, wherein each facet of themulti-faceted image comprises a plurality of pixels from a subset of thewide-angle image.
 17. The vehicle of claim 15, wherein the controller isfurther configured to receive the wide-angle image from a camera. 18.The vehicle of claim 17, further comprising the camera.
 19. The vehicleof claim 18, wherein the camera comprises a fisheye camera.
 20. Thevehicle of claim 15, wherein the controller is further configured todetermine the mapping based according to:u _(F) =u _(F,0) +c _(F)θcos ϕ, andv _(F) =v _(F,0) +c _(F)θsin ϕ, where u_(F) and v_(F) are pixellocations of the pixels within the wide-angle image, c_(F), u_(F,0) andv_(F,0) are intrinsic camera parameters, θ=arccos(n_(F,Z)) and θ=arctan2(n_(F,Y), n_(F,X)), where n_(F,X), n_(F,Y), and n_(F,Z) comprise arespective pixel unit vector corresponding to a Cartesian coordinatesystem.